Nonlinear Activation Function Research Articles

Fixed-time solution of inequality constrained time-varying linear systems via zeroing neural networks

Zeroing neural network (ZNN) has been explored and applied in a variety of time-varying problems solving. However, ZNN models for inequality constrained time-varying linear equation (ICTVLE) solving are rarely reported. Generally, numerous practical problems can be mathematically modeled as ICTVLE problems. Motivated by the above mentioned issue, a nonlinear tunable activation function activated ZNN (NTAF-ZNN) model is constructed to effectively solve ICTVLE problems. To further improve the performances of NTAF-ZNN model, a fuzzy nonlinear tunable activation function (FNTAF) is also designed. Based on the FNTAF, a fuzzy nonlinear tunable activation function-based ZNN (FNTAF-ZNN) model is realized. Firstly, the fixed-time convergence property and disturbance rejection ability of the NTAF-ZNN and FNTAF-ZNN models are proved by theoretical analysis. Then, comparative simulation results of the two models with other existing ZNN models for solving the ICTVLE problem are provided to further verify their robustness and effectiveness. Additionally, the two models and other existing ZNN models are also employed to realize inequality constrained manipulator trajectory tracking under ideal and noisy environments for further comparisons. Finally, the superior performances of the FNTAF-ZNN model for manipulator trajectory tracking are further verified by physical experiment results.

Activation Functions for Neural Networks: Application and Performance-based Comparison

Past decade has seen explosive growth of Deep Learning (DL) algorithms based on Artificial Neural Networks (ANNs) and its applications in vast emerging domains to solve real world complex problems. The DL architecture uses Activation Functions (AFs), to perform the task of finding relationship between the input feature and the output. Essential building blocks of any ANN are AFs which bring the required non-linearity of the output in the Output layer of network. Layers of ANNs are combinations of linear and nonlinear AFs. Most extensively used AFs are Sigmoid, Hyperbolic Tangent (Tanh), Rectified Linear Unit (ReLU) etc to name a few. Choosing an AF for a particular AF depends on various factors such as Nature of Application, Design of ANN, Optimizers used in the network, Complexity of Data etc. This paper presents a survey on most widely used AFs along with the important consideration while selecting an AF on a specific problem domain. A broad guideline on selecting an AF based on the literature survey has been presented to help researchers in employing suitable AF in their problem domain.

Protein function prediction through multi-view multi-label latent tensor reconstruction

BackgroundIn last two decades, the use of high-throughput sequencing technologies has accelerated the pace of discovery of proteins. However, due to the time and resource limitations of rigorous experimental functional characterization, the functions of a vast majority of them remain unknown. As a result, computational methods offering accurate, fast and large-scale assignment of functions to new and previously unannotated proteins are sought after. Leveraging the underlying associations between the multiplicity of features that describe proteins could reveal functional insights into the diverse roles of proteins and improve performance on the automatic function prediction task.ResultsWe present GO-LTR, a multi-view multi-label prediction model that relies on a high-order tensor approximation of model weights combined with non-linear activation functions. The model is capable of learning high-order relationships between multiple input views representing the proteins and predicting high-dimensional multi-label output consisting of protein functional categories. We demonstrate the competitiveness of our method on various performance measures. Experiments show that GO-LTR learns polynomial combinations between different protein features, resulting in improved performance. Additional investigations establish GO-LTR’s practical potential in assigning functions to proteins under diverse challenging scenarios: very low sequence similarity to previously observed sequences, rarely observed and highly specific terms in the gene ontology.ImplementationThe code and data used for training GO-LTR is available at https://github.com/aalto-ics-kepaco/GO-LTR-prediction.

ReActHE: A homomorphic encryption friendly deep neural network for privacy-preserving biomedical prediction

The growing distribution of deep learning models to individuals’ devices on sensitive healthcare data introduces challenging privacy and security problems when computation is being operated on an untrusted server. Homomorphic encryption (HE) is one of the appropriate cryptographic techniques to provide secure machine learning computation by directly computing over encrypted data, so that allows the data owner and model owner to outsource processing of sensitive information to an untrusted server without leaking any information about the data. However, most current HE schemes only support limited arithmetic operations, which significantly hinder their applications to implement a secure deep learning algorithm, especially on the nonlinear activation function of a deep neural network. In this paper, we develop a novel HE-friendly deep neural network, named REsidue ACTivation HE (ReActHE), to implement a precise and privacy-preserving algorithm with a non-approximating HE scheme on the activation function. We consider a residue activation strategy with a scaled power activation function in a deep neural network for HE-friendly nonlinear activation. Moreover, we propose a residue activation network structure to constrain the latent space in the training process to alleviate the optimization difficulty. We comprehensively evaluate the proposed ReActHE method using various biomedical datasets and widely-used image datasets. Our results demonstrate that ReActHE outperforms other alternative solutions to secure machine learning with HE and achieves low approximation errors in classification and regression tasks.

Time delay reservoir computing with a silicon microring resonator and a fiber-based optical feedback loop

Silicon microring resonators serve as critical components in integrated photonic neural network implementations, owing to their compact footprint, compatibility with CMOS technology, and passive nonlinear dynamics. Recent advancements have leveraged their filtering properties as weighting functions, and their nonlinear dynamics as activation functions with spiking capabilities. In this work, we investigate experimentally the linear and nonlinear dynamics of microring resonators for time delay reservoir computing, by introducing an external optical feedback loop. After effectively mitigating the impact of environmental noise on the fiber-based feedback phase dependencies, we evaluate the computational capacity of this system by assessing its performance across various benchmark tasks at a bit rate of few Mbps. We show that the additional memory provided by the optical feedback is necessary to achieve error-free operation in delayed-boolean tasks that require up to 3 bits of memory. In this case the microring was operated in the linear regime and the photodetection was the nonlinear activation function. We also show that the Santa Fe and Mackey Glass prediction tasks are solved when the microring nonlinearities are activated. Notably, our study reveals competitive outcomes even when employing only 7 virtual nodes within our photonic reservoir. Our findings illustrate the silicon microring’s versatile performance in the presence of optical feedback, highlighting its ability to be tailored for various computing applications.

Applying green learning to regional wind power prediction and fluctuation risk assessment

Deep Learning (DL) models, such as Long Short-Term Memory (LSTM) and Gated Recurrent Units (GRU), have been widely used to predict the intermittency of wind power; however, the non-linear activation functions and backpropagation mechanisms in DL models increase computational complexity and energy consumption. This paper proposes a prediction model based on Green Learning (GL) to reduce energy consumption. The proposed GL model replaces the feature extraction of activation functions with a hybrid feature extraction approach combining categorical and numerical features. We also employ cluster centroids and quantile regression forest for classification/regression to eliminate the need for backpropagation in optimizing hyperparameters. Using Taiwan as a case study, this paper evaluates the risk of fluctuations in regional wind power generation in 2030. In simulations, the proposed GL model achieved excellent accuracy with energy consumption significantly lower than that of DL models. Our analysis also revealed that by 2030, fluctuations in wind power generation during the winter will exceed 40% of the peak supply capacity in the central region, indicating the need to enhance the resilience of regional power systems.

Broadband nonlinear modulation of incoherent light using a transparent optoelectronic neuron array

Nonlinear optical processing of ambient natural light is highly desired for computational imaging and sensing. Strong optical nonlinear response under weak broadband incoherent light is essential for this purpose. By merging 2D transparent phototransistors (TPTs) with liquid crystal (LC) modulators, we create an optoelectronic neuron array that allows self-amplitude modulation of spatially incoherent light, achieving a large nonlinear contrast over a broad spectrum at orders-of-magnitude lower intensity than achievable in most optical nonlinear materials. We fabricated a 10,000-pixel array of optoelectronic neurons, and experimentally demonstrated an intelligent imaging system that instantly attenuates intense glares while retaining the weaker-intensity objects captured by a cellphone camera. This intelligent glare-reduction is important for various imaging applications, including autonomous driving, machine vision, and security cameras. The rapid nonlinear processing of incoherent broadband light might also find applications in optical computing, where nonlinear activation functions for ambient light conditions are highly sought.

Design and Analysis of a Novel Distributed Gradient Neural Network for Solving Consensus Problems in a Predefined Time.

In this article, a novel distributed gradient neural network (DGNN) with predefined-time convergence (PTC) is proposed to solve consensus problems widely existing in multiagent systems (MASs). Compared with previous gradient neural networks (GNNs) for optimization and computation, the proposed DGNN model works in a nonfully connected way, in which each neuron only needs the information of neighbor neurons to converge to the equilibrium point. The convergence and asymptotic stability of the DGNN model are proved according to the Lyapunov theory. In addition, based on a relatively loose condition, three novel nonlinear activation functions are designed to speedup the DGNN model to PTC, which is proved by rigorous theory. Computer numerical results further verify the effectiveness, especially the PTC, of the proposed nonlinearly activated DGNN model to solve various consensus problems of MASs. Finally, a practical case of the directional consensus is presented to show the feasibility of the DGNN model and a corresponding connectivity-testing example is given to verify the influence on the convergence speed.

Parametric RSigELU: a new trainable activation function for deep learning

Activation functions are used to extract meaningful relationships from real-world problems with the help of deep learning models. Thus, the development of activation functions which affect deep learning models’ performances is of great interest to researchers. In the literature, mostly, nonlinear activation functions are preferred since linear activation functions limit the learning performances of the deep learning models. Non-linear activation functions can be classified as fixed-parameter and trainable activation functions based on whether the activation function parameter is fixed (i.e., user-given) or modified during the training process of deep learning models. The parameters of the fixed-parameter activation functions should be specified before the deep learning model training process. However, it takes too much time to determine appropriate function parameter values and can cause the slow convergence of the deep learning model. In contrast, trainable activation functions whose parameters are updated in each iteration of deep learning models training process achieve faster and better convergence by obtaining the most suitable parameter values for the datasets and deep learning architectures. This study proposes parametric RSigELU (P+RSigELU) trainable activation functions, such as P+RSigELU Single (P+RSigELUS) and P+RSigELU Double (P+RSigELUD), to improve the performance of fixed-parameter activation function of RSigELU. The performances of the proposed trainable activation functions were evaluated on the benchmark datasets of MNIST, CIFAR-10, and CIFAR-100 datasets. Results show that the proposed activation functions outperforms PReLU, PELU, ALISA, P+FELU, PSigmoid, and GELU activation functions found in the literature. To access the codes of the activation function; https://github.com/serhatklc/P-RsigELU-Activation-Function.

Delay-dependent Lurie–Postnikov type Lyapunov–Krasovskii functionals for stability analysis of discrete-time delayed neural networks

This paper addresses the influence of time-varying delay and nonlinear activation functions with sector restrictions on the stability of discrete-time neural networks. Compared to previous works that mainly focuses on the influence of delay information, this paper devotes to activation nonlinear functions information to help compensate the analysis technique based on Lyapunov–Krasovskii functional (LKF). A class of delay-dependent Lurie–Postnikov type integral terms involving sector constraints of nonlinear activation function is proposed to complement the LKF construction. The less conservative criteria for the stability analysis of discrete-time delayed networks is given by using improved LKF. Numerical examples show that conservatism can be reduced by the delay-dependent integral terms involving nonlinear activation functions.

All-optical neural network nonlinear activation function based on the optical bistability within a micro-ring resonator

Training all-optical neural networks in itself remains an unresolved problem, and the challenges compound when the problem is turned into the hardware implementations. In this paper, we propose a nonlinear activation function based on optical bistability within a micro-ring resonator (MRR), achieving threshold control without external modulation. Furthermore, a convolutional neural network similar to the Le-Net-5 architecture is designed, in which all nonlinear activation functions are composed of optical bistable hysteresis loop. The numerical simulation results demonstrate that the recognition rate on the Fashion-MNIST dataset can achieve 91.3%, which means that the optical neuromorphic computation can be implemented by utilizing the nonlinear optical effects themselves in the all-optical hardware. Such a scheme promises access to the all-optical neural network training in the optical hardware environment compared to numerical activation functions.

Neural Generalization of Multiple Kernel Learning

Multiple Kernel Learning (MKL) is a conventional way to learn the kernel function in kernel-based methods. MKL algorithms enhance the performance of kernel methods. However, these methods have a lower complexity compared to deep models and are inferior to them regarding recognition accuracy. Deep learning models can learn complex functions by applying nonlinear transformations to data through several layers. In this paper, we show that a typical MKL algorithm can be interpreted as a one-layer neural network with linear activation functions. By this interpretation, we propose a Neural Generalization of Multiple Kernel Learning (NGMKL), which extends the conventional MKL framework to a multi-layer neural network with nonlinear activation functions. Our experiments show that the proposed method, which has a higher complexity than traditional MKL methods, leads to higher recognition accuracy on several benchmarks.

Personalized POI Recommendation Using CAGRU and Implicit Semantic Feature Extraction in LBSN

With the intention of addressing the concern that existing point of interest recommendation methods fail to fully utilize the auxiliary information of the point of interest, from which it is challenging to extricate a substantial quantity of deeper feature information, a personalized point of interest (POI) recommendation model using Context-Aware Gated Recurrent Unit (CAGRU) and implicit semantic feature extraction was proposed. First, the check-in data is divided into five tags, and the continuous geographical location check-in data and time data are discretized. Then, the CAGRU was used to obtain the POI check-in features. Finally, the time sequence location information, user information and target location information are transformed through the nonlinear activation function to obtain the score of each location in the data set as the next POI location, and the Top-N recommendation is generated through the score. Experiments indicated that the results of the suggested method were better than the comparative methods.

An effective hybrid particle swarm—artificial neural network optimization for predicting green bio-fiber mechanical characteristics and optimizing biomaterial performance

Natural fiber-reinforced composites are currently utilized in several applications due to worldwide environmental and cost concerns. However, these composites have production challenges such as poor reinforcement-matrix adhesion, that sophisticates the prediction of their mechanical properties. This study presents a novel, robust hybrid particle swarm—artificial neural network optimization (PSO-ANN) methodology to assess and create accurate predictions of the green bio-fibers to optimize and improve the mechanical features of biomaterials for green bio-products instead of performing tedious experimental works. As the mechanical qualities of green bio-fibers might differ from one fiber to another due to several interacted parameters, high complexity in predicting the bio-fiber capabilities exists. Therefore, this work utilizes suitable methods with a non-linear activation function to predict the mechanical characteristics of natural fibers that allow the researchers to improve the choices of natural fibers for biomaterials on the basis of cellulose content, the microfibrillar angle, and the diameter of natural fibers, decreasing the duration of the process required to characterize materials experimentally. The reliability of the introduced PSO-ANN model was verified by the investigations of the fiber’s tensile stress and Young’s modulus. Results showed that the presented model is capable of consistently and accurately monitoring the mechanical performance to a large degree, in comparison with experimental results. This in fact would facilitate and simplify the process of selecting the best natural fiber composites, which speeds up the experimental characterization phase and improves energy efficiency in the process of converting energy into monetary income, which would have ramifications for both economies and ecosystems. The anticipated method would also boost scientific evaluation of green fibers, confirming their role as a replacement material for green product fulfillment in future eco-friendly manufacturing.

Zeroing Neural Network With Coefficient Functions and Adjustable Parameters for Solving Time-Variant Sylvester Equation.

To solve the time-variant Sylvester equation, in 2013, Li et al. proposed the zeroing neural network with sign-bi-power function (ZNN-SBPF) model via constructing a nonlinear activation function. In this article, to further improve the convergence rate, the zeroing neural network with coefficient functions and adjustable parameters (ZNN-CFAP) model as a variation in zeroing neural network (ZNN) model is proposed. On the basis of the introduced coefficient functions, an appropriate ZNN-CFAP model can be chosen according to the error function. The high convergence rate of the ZNN-CFAP model can be achieved by choosing appropriate adjustable parameters. Moreover, the finite-time convergence property and convergence time upper bound of the ZNN-CFAP model are proved in theory. Computer simulations and numerical experiments are performed to illustrate the efficacy and validity of the ZNN-CFAP model in time-variant Sylvester equation solving. Comparative experiments among the ZNN-CFAP, ZNN-SBPF, and ZNN with linear function (ZNN-LF) models further substantiate the superiority of the ZNN-CFAP model in view of the convergence rate. Finally, the proposed ZNN-CFAP model is successfully applied to the tracking control of robot manipulator to verify its practicability.

Chebyshev polynomial approximation in CNN for zero-knowledge encrypted data analysis

Integrating Deep Learning (DL) techniques in Convolutional Neural Networks (CNNs) with encrypted data analysis is an emerging field for enhancing data privacy and security. A significant challenge in this domain is the incompatibility of standard non-linear Activation Functions (AF) like Rectified Linear Unit (ReLU) and Hyperbolic Tangent (tanh) with Zero-Knowledge (ZK) encrypted data, which impacts computational efficiency and data privacy. Addressing this, our paper introduces the novel application of Chebyshev Polynomial Approximation (CPA) to adapt these AF to process encrypted data effectively. Utilizing the MNIST dataset, this paper conducted experiments with LeNet and various configurations of AlexNet, extending the range of the ReLU and tanH functions to optimize CPA. Our results reveal an optimal polynomial degree (α), with α = 10 for ReLU and between α = 10 and α = 15 for tanH, beyond which the benefits in accuracy plateau. This finding is crucial for ensuring the accuracy and efficiency of CNNs in processing encrypted data. This recommended study demonstrates that while the accuracy slightly decreases for plaintext data and more significantly for ciphertext data, the overall effectiveness of CPA in CNNs is maintained. This advancement enables CNNs to process encrypted data while preserving privacy and marks a significant step in developing privacy-preserving Machine Learning (ML) and encrypted data analysis.

Aspect-Aware Graph Attention Network for Heterogeneous Information Networks.

Graph Convolutional Networks (GCNs) derive inspiration from recent advances in computer vision, by stacking layers of first-order filters followed by a nonlinear activation function to learn entity or graph embeddings. Although GCNs have been shown to boost the performance of many network analysis tasks, they still face tremendous challenges in learning from Heterogeneous Information Networks (HINs), where relations play a decisive role in knowledge reasoning. What's more, there are multiaspect representations of entities in HINs, and a filter learned in one aspect do not necessarily apply to another. We address these challenges by proposing the Aspect-Aware Graph Attention Network (AGAT), a model that extends GCNs with alternative learnable filters to incorporate entity and relational information. Instead of focusing on learning the general entity embeddings, AGAT learns the adaptive entity embeddings based on prediction scenario. Experiments of link prediction and semi-supervised classification verify the effectiveness of our algorithm.

A Transistor Operations Model for Deep Learning Energy Consumption Scaling Law

Deep Neural Networks (DNN) has transformed the automation of a wide range of industries and finds increasing ubiquity in society. The high complexity of DNN models and its widespread adoption has led to global energy consumption doubling every 3-4 months. Current energy consumption measures largely monitor system wide consumption or make linear assumptions of DNN models. The former approach captures other unrelated energy consumption anomalies, whilst the latter does not accurately reflect nonlinear computations. In this paper, we are the first to develop a bottom-up Transistor Operations (TOs) approach to expose the role of non-linear activation functions and neural network structure. As there will be inevitable energy measurement errors at the core level, we statistically model the energy scaling laws as opposed to absolute consumption values. We offer models for both feedforward DNNs and convolution neural networks (CNNs) on a variety of data sets and hardware configurations - achieving a 93.6% - 99.5% precision. This outperforms existing FLOPs-based methods and our TOs method can be further extended to other DNN models.

A Method of Body Condition Scoring  for Dairy Cows Based on Lightweight  Convolution Neural Network

Highlights A cow body condition scoring model (ShuffleNet-SEH) based on a lightweight convolutional neural network using ShuffleNetV2 was proposed. Introducing the SE attention mechanism and H-Swish activation function enhances the feature extraction capability of the model. Achieved over 98% classification accuracy rate on the cow datasets. The proposed method can be applied to mobile devices. Abstract. Body condition scoring is an essential tool for nutrition management in large-scale dairy farming. The traditional method of manual scoring is inefficient and prone to subjective error. With the advancement of deep learning algorithms and machine vision technology, automatic scoring methods have been proposed. In response to the current problems of large parameter sizes and poor robustness in deep learning models, this study proposes a cow body condition scoring model (ShuffleNet-SEH) based on a lightweight convolutional neural network using ShuffleNetV2 as a foundation. Firstly, tail area images of cows were collected to construct a diverse dataset that includes complex variations such as motion blur and different levels of illumination. Secondly, the ShuffleNet-SEH model was developed by integrating a Squeeze-and-Excitation Networks (SENet) module into the main branch of ShuffleNetV2 after 1×1 convolution to enhance the representation ability of the network and improve its accuracy in identifying the cow body condition. Additionally, the nonlinear activation function ReLU in SENet was replaced by H-Swish to reduce the computational data required for the model on mobile devices, making it suitable for quantization operations. ShuffleNet-SEH was comprehensively evaluated, including ablation experiments, confusion matrix analysis, and feature map analysis. Multiple complex test sets were constructed for validation purposes. Throughout these evaluations, ShuffleNet-SEH exhibited remarkable performance. Furthermore, the efficacy and interpretability of ShuffleNet-SEH in cow body condition scoring tasks were further substantiated through meticulous feature map analysis. The findings underscore the robustness and reliability of ShuffleNet-SEH across diverse experimental scenarios and assessments. In the constructed motion blur test set, ShuffleNet-SEH demonstrated outstanding performance. Specifically, it achieved an accuracy rate of 98.2%, a precision rate of 98.5%, and a recall rate of 98.3%. These results represent significant improvements of 2.3%, 2.4%, and 3.2%, respectively, compared to the performance of the ShuffleNetV2 model. In addition, it is noteworthy that the ShuffleNet-SEH model has a size of approximately 5.64 MB, while the original ShuffleNet model has a size of 4.97 MB. This indicates a modest increase in model size of approximately 13.5% with the incorporation of the ShuffleNet-SEH enhancements. Moreover, superior overall performance was displayed by ShuffleNet-SEH model compared to mainstream convolutional neural network classification models, including MobileNetV2, EfficientNetV1, and ConvNeXt. The success of ShuffleNet-SEH in accurately identifying cow body condition while maintaining a lightweight architecture makes it suitable for deployment on mobile devices and, thus, has the potential to promote the commercialization of cow body condition scoring. Keywords: Attention mechanism, Body condition score, Deep learning, ShuffleNetV2.

Brainwave implanted reservoir computing

This work aims to build a reservoir computing system to recognize signals with the help of brainwaves as the input signals. The brainwave signals were acquired as the participants were listening to the signals. The human brain in this study can be regarded as the assistant neural networks or non-linear activation function to improve the signal recognition. We showed that within the brainwave frequency ranges from 14 to 16, 20, 30, and 32 Hz, the mean squared errors of the input signal recognition were lower than those without brainwaves. This result has demonstrated that the reservoir computing system with the help of human responses can obtain more precise results.